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A Comparison  of  Information  Functions  of  Multiple-Choice 
And  Free-Response  Vocabulary  Items 


The  multiple-choice  item  format  used  by  most  group  tests  of  mental  ability 
allows  testees  to  obtain  correct  answers  by  guessing  when  they  do  not  know  the 
correct  answer.  This  adds  error  variance  to  test  scores  or,  in  terms  of  modern 
test  theory  (see  Lord  & Novick,  1968,  chaps.  16-20),  decreases  the  amount  of 
information  that  the  item  provides  about  ability  levels. 

Several  attempts  have  been  made  to  eliminate  guessing  by  making  its  effects 
less  attractive:  Formula  scoring  ( i.e .,  correction  for  guessing)  subtracts 

points  for  items  answered  incorrectly,  making  the  expected  gain  from  guessing 
negligible;  confidence  weighting  and  probabilistic  responding  strategies 
typically  use  Reproducing  Scoring  Systems  (Shuford,  Albert,  6 Massengill,  1966) 
which  cause  testees*  subjective  test  scores  {i.e.,  the  score  they  think  they 
will  get)  to  be  maximized  when  they  answer  honestly.  Another  scoring  technique 
attempts  to  eliminate  the  effects  of  guessing  by  simply  not  scoring  those  items 
on  which  a testee  is  likely  to  guess  (Waller,  1974).  These  approaches  and 
others  have  been  reviewed  by  Bejar  (1975) . 

The  research  reported  here  was  an  attempt  to  eliminate  the  effects  of 
guessing  by  making  it  virtually  impossible  to  obtain  a correct  answer  to  a 
question  solely  by  guessing.  This  was  done  by  administering  items  in  a free- 
response  format  in  which  testees  were  required  to  generate  their  own  response 
instead  of  choosing  from  several  alternatives  that  are  provided.  To  be  practical 
as  a group  testing  approach  {i.e.,  as  an  alternative  to  multiple-choice  items), 
these  items  had  to  be  administered  and  scored  by  a computer. 

The  question  of  interest  guiding  this  research  was,  therefore:  Will  the 

information  gained  due  to  the  elimination  of  guessing  using  the  free-response 
format  be  greater  than  the  information  lost  due  to  inefficiencies  in  the  machine- 
scoring  algorithm?  Inefficiencies  refer  to  things  such  as  a higher  probability 
of  errors  in  responses  {e.g.,  typing  errors)  due  to  the  more  complex  format,  and 
the  need  to  group  responses  into  categories  because  they  are  too  numerous  to 
handle  individually. 

The  answer  to  this  question  is  obviously  dependent  on  the  domain  of  ability 
being  tested.  There  is  practically  no  inefficiency  in  the  scoring  of  free- 
response  numerical  items;  thus,  these  items  do  not  provide  an  interesting  area 
of  study.  Vocabulary  items,  in  which  the  responses  are  English  words,  provide 
a more  interesting  area  of  study  because  information  will  be  lost  due  to 
misspelling,  categorizetion,  etc.  The  objective  of  this  study  was,  therefore, 
to  determine  if  a machine-scoring  algorithm  could  be  implemented  to  extract  more 
information  out  of  free-response  vocabulary  items  than  was  obtained  from  those 
administered  in  a multiple-choice  format. 

Method 

Purpose 

This  study  involved  a comparison  of  vocabulary  test  items  administered  in  a 
free-response  format  with  similar  vocabulary  items  administered  in  a multiple-choice 
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format  In  terms  of  the  amount  of  information  each  provided  regarding  a testee's 
level  of  ability.  Toward  that  end,  20  five-alternative  multiple-choice  items 
were  randomly  sampled  from  a 36-item  conventional  test  with  rectangularly 
distributed  item  difficulties  used  as  part  of  another  study.  These  items  were 
obtained  from  Educational  Testing  Service  and  had  been  used  in  their  SCAT  and 
STEP  tests;  the  items  had  thus  been  carefully  analyzed  and  were  good  multiple- 
choice  items.  The  stem  words  from  these  items  provided  the  stems  for  20 
free-response  items.  In  the  free-response  items,  testees  were  asked  to  respond 
with  a synonym  rather  than  Indicating  their  choice  of  multiple-choice  alternative. 

Testees 

In  order  to  provide  data  from  which  to  calibrate  these  items  and  thus  deter- 
mine their  information  functions,  testees  were  recruited  from  two  sections  of 
an  Introductory  psychology  class  at  the  University  of  Minnesota  consisting  pri- 
marily of  sophomores  from  the  College  of  Liberal  Arts.  Test  items  were  presented 
to  testees  via  cathode  ray  terminals  (CRTs)  interfaced  to  a Hewlett-Packard  9600E 
real-time  minicomputer  system.  Items  were  displayed  at  a rate  of  960  characters 
per  second  (almost  instantaneously)  beginning,  typically,  less  than  a second  after 
the  testee's  response  to  the  previous  item.  Testees  could  skip  items  by  typing 
in  a "?"  as  their  response  if  they  did  not  know  the  answer  and  chose  not  to  guess. 

Tests 

Both  the  20-item  free-response  test  and  the  20-item  multiple-choice  test 
were  administered  to  all  students.  In  ell  cases,  the  free-response  test  was 
administered  first  (following  other  multiple-choice  items  which  were  independent 
of  those  used  in  this  study)  followed  immediately  by  the  multiple-choice  test 
Imbedded  in  the  36-ltem  test.  The  multiple-choice  test  was  administered  second 
to  avoid  providing  the  testees  with  alternatives  to  use  in  the  free-response 
test  (since  all  the  free-response  items  were  present  as  stems  in  the  multiple- 
choice  test) . 


At  the  beginning  of  the  free-response  test,  each  testee  was  given  the 
following  instructions  on  the  CRT: 


Now  you  are  going  to  be  given  some  vocabulary  test  questions  which 
are  different  from  those  you've  answered  so  far.  These  questions  will 
not  require  you  to  choose  the  correct  response  from  a set  of  alternatives. 
Instead,  you  ore  to  type  in  a one-word  response.  The  computer  will  present 
a word  or  phrase  and  you  are  to  respond  by  typing , on  the  keyboard,  the 
single  word  that  is  most  alike  in  meaning  to  that  word  or  phrase. 

For  example % the  computer  might  present  the  word  "wealthy" , followed 
by  a question  mark.  If  you  thought  the  word  most  similar  to  "wealthy" 
was  "rich",  you  would  respond  by  typing  the  word  "rich"  after  the  question 
mark. 

When  the  word  "wealthy"  appears,  type  in  the  word  "rich"  after  the 
question  mark  to  show  that  you  understand  the  instructions . If  you  do  not 
understand  the  instructions , type  in  a question  mark.  Remember  that  you 
must  always  press  the  return  key  when  you  have  finished  typing  in  your 
response. 
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If  testees  failed  to  enter  the  word  "rich",  they  were  given  instructions 
to  call  the  proctor  for  assistance.  Otherwise,  the  following  message  was 
presented: 

New  you  are  ready  to  take  this  part  of  the  test.  If  you  do  not 
know  the  answer  to  a question,  type  in  a question  mark  to  skip  that  question. 

Type  in  "Go"  to  start  the  test. 

At  the  end  of  the  free-response  test,  the  following  transition  back  to 
more  multiple-choice  items  was  made: 

Thank  you.  That  was  the  last  question  of  that  type.  Now,  for  the 
last  part  of  today's  test  we  are  going  to  give  you  some  more  multiple-choice 
vocabulary  questions.  Some  of  these  questions  will  contain  the  same  words 
you  encountered  in  the  section  you  just  finished.  These  questions  are  being 
repeated  so  that  we  can  compare  how  well  you  do  when  we  don't  provide  the 
alternatives.  When  answering  these  questions,  choose  the  best  alternative 
from  the  five  available  even  if  none  of  these  alternatives  seem  as  good  as 
your  own  response. 

Type  in  "Go"  to  start  this  section  of  the  test. 

Following  the  36-item  multiple-choice  test,  other  testing  continued. 

Analysis  of  Free  Responses 

Data  were  collected  from  660  testees.  More  than  60  formally  different 
responses  ( i.e .,  words  that  were  not  exactly  the  same)  were  obtained  for  each 
of  the  20  free-response  stems.  Due  to  computer-program  limitations,  these 
raw  responses  were  reduced  to  the  60  most  frequent  responses.  To  complete  the 
analysis,  this  number  had  to  be  further  reduced — immediately  to  nine  categories 
and  ultimately  to  six  (again  because  of  program  limitations) . 

The  many  different  responses  generated  by  the  testees  consisted  of  four 
distinct  types:  1)  frequent  responses,  both  correct  and  incorrect;  2)  misspell- 

ings of  the  frequently  used  words;  3)  variations  on  the  roots  of  these  words 
(e.g.,  "loyal"  and  "loyalty”);  and  4)  infrequent  responses  not  Included  in  Type 
2 or  3.  To  reduce  the  number  of  categories,  all  responses  were  first  ranked  in 
order  of  their  frequency.  Then  the  most  frequent  response  and  other  formally 
similar  responses  (i.e.,  composed  of  a similar  string  of  letters)  were  grouped 
into  a category,  using  a formal  similarity  detection  technique  based  on 
Alberga's  (1967)  "algorithm  25"  with  the  recommended  threshhold  of  .12.  This 
algorithm  and  threshhold  proved  best,  out  of  a field  of  65  in  a simulation 
by  Alberga,  for  recognizing  misspellings  of  target  words.  Visual  inspection  of 
the  present  data  suggested  that  this  technique  did  Indeed  recognize  misspelled 
words.  In  this  study,  the  technique  was  used  to  detect  both  misspellings  and 
variations  of  roots.  The  Fortran  IV  subroutine  used  is  Included  in  Appendix  A. 
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Response  clustering  continued  until  eight  formally  similar  clusters  (usually 
Including  one  "omit"  category)  had  been  Identified.  A final  "other"  category, 
containing  all  other  responses,  completed  the  nine  categories  manageable  by  the 
programs  used  at  this  point  In  the  analysis. 

These  nine  categories  were  then  clustered  on  the  basis  of  judged  semantic 
similarity  In  an  attempt  to  have  at  least  30  responses  in  each  category  (a  number 
arbitrarily  chosen  as  a minimum  for  acceptable  calibration  of  the  category) . 

Words  that  were  semantically  very  similar  were  clustered;  infrequently  used  and 
definitely  incorrect  words,  when  present,  were  clustered  with  the  "omit"  category; 
Infrequent  responses  not  semantically  similar  to  any  of  the  other  categories  and 
not  completely  incorrect  were  clustered  with  the  "other"  category.  The  nine 
initial  categories  and  their  ultimate  classifications  are  presented  In  Appendix 
Table  B— 1. 

An  attempt  was  made  to  semantically  cluster  the  alternatives  of  the  multiple- 
choice  items,  but  no  semantic  similarity  was  found.  Alternatives  were  thus 
either  allowed  to  stand  alone  if  their  frequency  of  endorsement  was  high  enough 
or  were  grouped  Into  an  "other"  category  if  it  was  not.  The  multiple-choice 
alternatives  and  their  ultimate  classifications  are  presented  In  Appendix 
Table  B-2. 

Item  Calibration 

The  Item  responses  thus  categorized  were  then  calibrated  according  to  Bock's 
(1972)  polychotomous  logistic  model  using  the  program  LOGOG  (Kolakowskl  & Bock, 
1973).  Bock's  program  yields  two  parameters,  a and  <?,  for  each  response  category 
of  each  test  Item.  It  should  be  noted  that  the  a and  a parameters  have 
different  interpretations  than  the  a and  o parameters  usually  calculated  in  item 
characteristic  curve  theory  ( e.g Lord  & Novick,  1968). 

Let  i and  j index  the  I- J categories  of  a given  item.  The  probability 
(P^.)  of  a testee  endorsing  category  j as  a function  of  ability  (8)  Is: 

I 

P.  - «*J/  l ezi  mi 

0 iml 

where 

\ " °i  * ai 6 [2] 

and  a ^ and  represent  the  parameters  corresponding  to  category  i of  the  item. 

The  parameters  and  a^,  and  the  function  eZi  might  be  psychologically 

m 0 

interpreted  as  follows:  e v can  be  thought  of  as  the  attractiveness  of  response 

alternative  (or  category)  i of  the  item  as  perceived  by  die  testee.  As  e*1  gets 
larger,  category  i becomea  more  attractive  to  the  testee.  But  in  deciding  which 
alternative  to  endorse,  the  testee  must  also  consider  the  attractiveness  of  the 
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other  response  alternatives  for  that  item.  Thus,  according  to  the  model,  the 
testee's  probability  of  endorsing  a category  (Equation  1)  Is  equal  tc  the 
attractiveness  of  a given  category  divided  by  the  sum  of  the  attractiveness  of 

all  the  categories.  The  function,  e3^,  is  a monotonic  increasing  function  of 
2^,  and  2^  Is  either  a monotonic  increasing  or  monotonic  decreasing  function  of 

ability  (0)  depending  on  the  sign  of  the  parameter  a.  (Equation  2).  As  a. 

increases  in  absolute  value,  a given  change  in  ability  is  associated  with  a 
larger  change  in  attractiveness.  Thus,  a.  may  be  thought  of  as  an  index  of 

category  discriminating  power.  If  a.  is  positive,  attractiveness  increases 

with  increasing  ability,  can  be  interpreted  as  an  attractiveness-biasing 

parameter.  As  a category’s  a parameter  increases,  the  attractiveness  of  the 
category  gets  larger  at  all  levels  of  ability.  When  0-0,  the  attractiveness 
of  the  categories  is  ranked  in  order  of  their  a parameters. 

Calculation  of  Information 

Using  the  category  parameters  obtained  from  the  item  calibration,  item 
information  was  calculated  from  Same J lma ' s (1969,  chap.  6)  general  equations. 
The  first  and  second  derivatives  of  the  probability  function  are  given  by: 
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3 V 


i~l 
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and 
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J t- 

[4] 


I/?. information  provided  about  ability  by  an  item  as  a function  of  ability 
T(0)  is  then  given  by  Equation  5 where: 

t5] 


J 


1(0)  - E (p'Mp.-pV) 
3=1  J J J 


Item  information  values  were  calculated  from  Equation  5 for  each  multiple- 
choice  item  and  each  free-response  item  at  25  points  along  the  ability  continuum 
between  0--3  to  0-f3.  For  each  response  format  at  each  of  the  25  ability  levels, 
information  values  for  each  of  the  items  were  added  together  to  yield  a test 
information  value,  and  a smoothed  curve  was  passed  through  these  values  to  yield 
the  two  test  information  functions. 


Item  Parameters 


Results 


parameters,  a and  o,  along  with  chi-square  goodness -of -fit  statis- 
tics are  shown  in  Table  1 (free-response)  and  Table  2 (multiple-choice).  Free- 
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Parameters  and  Tests  of  Fit  for  Multiple-Choice  Items 
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response  items  11  and  19  (Table  1)  showed  lack  of  fit  with  the  logistic  model 
significant  at  p<.0 5,  but  the  other  items  exhibited  no  significant  lack  of  fit. 
Multiple-choice  items  12,  17,  18,  19,  and  20  (Table  2)  showed  significant  lack 
of  fit  with  the  model.  This  lack  of  fit  is  probably  an  effect  of  guessing, 
which  Bock's  model  assumes  does  not  occur.  Although  testees  were  given  the 
opportunity  to  omit  items,  it  is  likely  that  some  guessed  anyway;  items  17 
through  20  were  the  most  difficult  items  and  thus  most  likely  to  elicit  guessing 
behavior.  Item  18  showed  profound  lack  of  fit  and  had  extremely  unrealistic 
item  parameters  in  the  multiple-choice  format.  It  was  therefore  excluded  from 
further  analyses  in  both  the  multiple-choice  and  free-response  formats. 

Information 

Smoothed  test  Information  functions  for  both  the  multiple-choice  and  free- 
response  tests  are  shown  in  Figure  1.  Test  information  values  for  both  tests 
are  included  in  Appendix  Table  B-3. 


Figure  1 

Test  Information  Functions  for  Two  19-Item  Tests 


ABILITY  LEVEL  (0) 


Figure  1 shows  that  both  tests  were  too  easy  for  the  population  to  which 
they  were  applied  in  this  study  because  they  provided  maximal  information  at 
about  8*1.5;  assuming  6 distributed  normally  with  a mean  of  zero  and  a standard 
deviation  of  1.0,  the  teat  information  function  should  have  peaked  at  0-0  to 
provide  the  highest  reliability  coefficient. 
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But  more  importantly.  Figure  1 shows  that  while  items  administered  in  the 
two  response  formats  provided  equivalent  amounts  of  Information  near  ability 
levels  where  the  information  function  peaked,  the  free-response  items  yielded 
more  Information  at  the  higher  ability  levels.  Simply  removing  the  effects  of 
guessing  should  result  in  increased  information  at  low-ability  levels  rather 
than  at  high  ones  because  the  effects  of  guessing  are  greater  at  low-ability 
levels  (see  Figure  20.4.2  in  Lord  & Novick,  1968).  But  the  free-response 
format  involves  a recall  task  (rather  than  a recognition  task  as  in  the  multiple- 
choice  test  format),  and  this  probably  made  the  items  more  difficult,  thus 
shifting  the  information  function  to  the  right. 

These  information  functions  should  be  viewed  further  in  light  of  the  ease 
of  construction  of  the  two  types  of  items.  The  multiple-choice  items  were 
undoubtedly  written  by  professional  item  writers  at  ETS,  selected  for  their 
ability  to  discriminate  ability  levels,  and  were  designed  with  well- functioning 
distractors.  Beyond  selection  of  the  stems  (which,  in  this  study,  were  fixed 
by  the  multiple-choice  items) , the  20  free-response  items  required  only  a 
few  minutes  of  computer  time  to  score  and  about  one  hour  to  develop.  With 
further  research,  designed  to  develop  guidelines  for  selection  of  good  stems, 
good  free-response  vocabulary  items  would  be  much  easier  to  produce  than  are 
mu'  iple-choice  items. 

Conclusions 

This  study  has  shown  that  vocabulary  items  presented  in  a free-response 
format  can  provide  more  information  than  similar  items  presented  in  a multiple- 
choice  format.  There  are  two  probable  sources  of  this  superiority.  First, 
obtaining  a correct  answer  by  guessing  is  not  possible  using  the  free-response 
format,  and  information  lost,  due  to  the  uncertainty  about  whether  testees 
answered  correctly  because  they  knew  the  answer  or  because  they  guessed,  is 
recovered.  Secondly,  more  latitude  in  degree  of  correctness  is  present  in 
free-response  items  than  is  typically  present  in  multiple-choice  items  and  a 
testee's  degree  of  partial  knowledge  is  easier  to  aitsess.  These  advantages 
apparently  overshadowed  any  deficiencies  present  in  the  machine-scoring 
algorithm  used  in  this  study. 

This  research  was  designed  as  a demonstration  that  the  free-response  format 
scored  by  a computer  is  more  informative  than  the  multiple-choice  format.  It 
was  not  a far-reaching  comparison  with  all  other  potential  response  formats. 

Future  research  should  compare  free-response  items  with  other  formats  such  as 
a confidence-weighting  format  or  a probabilistic  format.  It  should  also  compare 
them  with  multiple-choice  items  having  more  alternatives  and/or  wrong  alternatives 
graded  in  difficulty  rather  than  all  completely  incorrect.  (Although  scored  as  if 
they  were  gradable  in  this  study,  the  alternatives  of  the  items  used  were  not 
designed  to  be  graded  in  difficulty.)  Future  research  should  also  investigate 
the  effects  of  various  techniques  of  semantic  clustering  and  detection  of  formal 
similarity . The  techniques  used  in  this  study  were  probably  not  optimal  for 
extracting  maximal  information  from  the  free-response  items  (i.e.,  the  clustering 
and  similarity  detection  were  clinical  in  nature  and  were  not  explicitly 
designed  to  extract  maximum  information  from  the  items),  and  better  techniques 
should  produce  results  even  more  favorable  to  the  free-response  format. 
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APPENDIX  A 

A Fortran  Subroutine  for  Assessing 
the  Formal  Similarity  of  Two  Words 
(from  Alberga,  1967) 


SULR0''T1NE  MATCH  ( 1 TARG.  I TEST.N TaRG, NT.  SI  M7.iL) 

DIMENSION  I TARG (NTARG). I TEST! NT) .COIN! 20. 20). I COL! 20) 

C ROUTINE  DETERMINES  SIMILARITY  bETVEEN  TW0  VORLS 

C ACC0Rt INC  T0  ALGORITHM  25  REPORTED  in  ALbERGA.  1967. 

C COMMUNICATIONS  OF  ACM. 

C PAr.AMETF.r«  j 

C ITARG  • ARRAY  CONTAINING  TARGET  WORD  - ONE  CaaRaCTEFv  PER 

C COMPUTER  WORE#  RIGHT  JUSTIFIED.  ZER0  F1LLEI 

C 1 TEST  • ARRAY  CONTAINING  TEST  WORD 

C NTARG  ■ NUKfeER  OF  CHARACTERS  IN  TARGET  WORD 

C NT  » NUMLER  OF  CHARACTERS  IN  TEST  WORL 

C SIM7AL  « RETURNET  SIMILARITY  VALUE 

C 

10  TO  20  1-1.20 

icoLtn-o 

CO  20  J-l ,20 
COIN! I , J) -0.0 
20  CONTI  .WE 

C-NTARG 

T-NT 

C FILL  COINCIDENCE  MaTRIX  WITH  ROOF  WEIGHTS 

100  DO  110  1-1, NTARG 
t»  I 10  J-l .NT 

IF  (ITARG(I)  .NE.  I TEST! J))  GO  TO  110 
Cl ST-ADSl FLOAT! I -l J/CC-l .0 > -FL0aT( J - l )/(  T-l .0) ) 

C91N(1,J)«I .0-CIST 
110  CONTIN'T 

C SELECT  ELEMENTS  ACCORDING  TO  S.'.YC  ALGORITHM 

200  CO  230  I-l, NTaRG 
TEST-- I .0 
LOC-O 

10  210  J-l .NT 

IF  (COIN(l.J)  .LT.  TEST  .OR.  1C0L(J>  .EG.  II  GO  TO  210 
TEST-COIN! l.J) 

loc-j 

210  CONTINUE 

IF  ILOC  .GT.  01  1 COL! LOC > -1 
CO  220  J- 1 .NT 
IF  ! J .EC.  LOC)  GO  TO  220 
COIN! I , J ) -0 . 0 
220  CONTINUE 
230  CONTINUE 

C S'*  ACCOSTING  TO  STRING  ALGORITHM 

S'IM-0.0 
J-l 

300  PRE7-0.0 

IF  ! J .CT.  NT)  GO  TO  400 
CO  310  1-1, NTARG 

IF  ! C01 N! I , J)  .GT.  0.0)  GO  TO  320 
310  CONTINUE 

J-J*l 
GO  TO  300 

320  PRE7-PRE7-C0IN! I » J) 

SUM-SUM-PRE7 
J-J  + l 
1-1*1 

IF  ! 1 .GT.  NTARG  -OR.  J .GT.  NT  .OR.  C01NU.J)  • LE.  G.U)  GO  TO  30U 
GO  TO  320 

C NORMALIZE  SUM  FOR  SIMILARITY  VALUE 

400  CONTINUE 

IMAX-NTARG 

IF  (NT  .GT.  IMAX)  IMaX-NT 
XMAX ■ 1 MAX 

SI  M7AL- S UM  / ( 0 . 5*  t XMAX -XMAX  *XMAX  ) > 

RETtrN 

END 
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Table  »-l 

cyatoriw  ctat>lltf».u«»8«  to  tuadaaaan  la 


Category 

No. 

Stan 

1 

2 

1 

A 

5 

6 

1 

T0LKIA1LE 

burabli 

(no  raaponaa) 

acceptable 

okay 

patient 
unde  rn  tend 

etandabla 

withstand 

(other) 

2 

ALLEGIANCE 

loyalty 

(no  raaponaa) 

Had*# 

rapport 

alliance 

falthfulnaaa 

patriot lea 

honor 

(other) 

3 

CATASTVOPHE 

disaster 

(no  re opens*) 
tarrlbla 

accident 

tragedy 

) 

crlala 

mess 

chaos 

(other) 

4 

DIMINISH 

laaaon 

decrease 

•brink 

fad* 

reduce 

and 

disappear 

•nailer 

(othar) 

i 

IWIACULATE 

clean 

epotlaaa 

(no  raaponaa) 

huge 

perfect 

purs 

holy 

aaat 

(other) 

t 

CHBONIC 

racurrlac 

habitual 

(no  raaponaa) 
serious 

conatant 

alvaya 

lasting 

peralatant 

(othar) 

7 

HOMAGE 

ntptct 

honor 

(no  raaponaa) 

tribute 

allegiance 

praise 

worship 

adoration 

(othar) 

B 

FLOG 

whip 

boat 

laah 

apank 

flafallata 

(no  response) 

ala take 

punish 

(othar) 

9 

AWOH 

hata 

detest 

dislike 

•hun 

(no  raaponaa) 
• tick 

feer 

hide 

(othar) 

10 

IMPEDE 

111 

(no  raaponaa) 

atop 

block 

obstruct 

prevent 

(othar) 

11 

UrilMAHD 

•cold 

•daonlsh 

reprove 

(no  raaponaa) 

punish 

discipline 

cornet 

(othar) 

12 

ADAGE 

•aylna 

proverb 

(no  raaponaa) 
addition 

atory 

cliche 

phraaa 

tala 

(othar) 

11 

ACCLAIM 

pralaa 

(no  raaponaa) 

racognltion 

iiii 

(othar) 

M 

QUALM 

doubt 

mlsalvlng 

raaarvatlon 

(no  raaponaa) 

fight 

irpaiat 

fiat 

worry 

(other) 

11 

on 

sphere 

ball 

(loba 

(no  raaponaa) 

•ja 

path 

circle 

round 

(othar) 

U 

ALLOT 

•Uocata 

•aatfa 

(no  raaponaa) 

distribute 

divide 

ration 

give 

•lion 

(other) 

17 

raapand 

(no  response) 
accept 

rocognlao 

notice 

uadaratand 

know 

(othar) 

It 

mum 

pnclfp 

cate 

(no  raaponaa) 

soothe 
cent  art 
quiet 

subdue 

(othar) 

It 

fflMTE 

tala 

ewlat 

paacafal 

(no  raaponaa) 

drag 

trangnlllta 

sleep 

re  lea 

(othar) 

20 

ncwun 

n stary 
financial 

(no  raaponaa) 

am  U 

tlakr 

pacullar 

strange 

different 

(othar) 

13 


C«ti|orln  and  Claeelf lest ions 


Table  R-2 

far  aach  of  Twenty  Multiple-Choice  Itei 


1:  1 


Item 

Cnmory 

Ho. 

See* 

i 

2 

3 

4 

3 

6 

1 

TOLERABLE 

bearable 

free 

flexible 
open-minded 
inferior 
(no  response; 

2 

ALLEGIANCE 

loyalty 

reading 

legibility 
protect ion 
fighting  unit 
(no  naponae ) 

3 

CATASTROPHE 

calamity 

celebration 

charity 

temlnatlon 

prophecy 

(no  naponae) 

4 

DIMINISH 

lessen 

flatten 

default 

undermine 

finish 

(no  naponae) 

3 

IMACULATE 

apotleas 

fashionable 

distinguished 

tardy 

powerless 

(no  naponae ) 

• 

6 

CHRONIC 

constant 

cowardly 

recorded 

weak 

grouchy 

(no  naponae) 

7 

HOMAGE 

reverence 

abode 

baseness 

food 

manhood 

(no  naponae) 

• 

FLOG 

bast 

stun 

tread 

bother 

soak 

(no  naponae) 

t 

ABHOR 

dataat 

frighten 

Accept 

urge 

release 

(no  naponae ) 

10 

IMPEDE 

oba tract 

auauon 

betray 
go  by  foot 
interrogate 
(no  naponae) 

11 

REPRIMAND 

rebuke 

refer  to  higher 
authority 

racall  by 
contrary 
orOar 

demand  repeatedly 
send  back 

(no  naponae  i 

12 

ADAGE 

proverb 

latar  years 

custom 

mental  weakness 
normal  condition 

(no  naoonae ' 

13 

ACCLAIM 

* 

flaunt 

indemnify 

oloct 

denounce 
(no  naponae ' 

14 

QUALM 

ol.';l*lo» 

foaling  of 

state  of  rest 

shudder 

duty 

(nc  naponae  ' 

15 

ORB 

afbart 

scepter 

dome 

track 

apur 

(no  naponae  f 

14 

ALLOT 

aaelgn 

permit 

Increase 
spend 
seclude 
(no  naponae) 

17 

ACKNOWLEDGE 

adult 

understand 

loan  (lowly 
onaaino  with 
can 

approve  of 
l nc  rrtponst) 

!• 

MOLLIFY 

appease 

KCtHftU 

lnd— ilfy 
PA*P«r 

revise 

(no  napewiaa) 

If 

20 

SEDATE 

PECUNIARY 

HlMf 

aiaarly 

aoatoO 

olC-foahlaooC 

frail 

(no  rooponotl 
goatltwto 

(nc 

■MMCAVp 

free went 

unueuel 

W&iiy  .. 

HU 

HHM 

EKEl&HMi 

i,  . 

HR  : O-yyi'' 

HSMfMMPVWM 
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Office  of  tke  Assistant  Secretary  of 
Defense,  K3KA 


Mr.  Samuel  Ball 
Educational  Testing  Service 
Frincctor,  I1J  03540 


fir  Force 

1 Research  Branch 
AFill’C/DPMVP 

Randolph  AFB,  TX  78148 

1 AfHRL/AS  (Dr.  G.  A.  Eckstrand) 
Wi  ignt- Patterson  AFB 
Ohio  45433 

1 Dr.  Marty  Rockway  (AFHRL/TT) 

lO"  -y  K> 

Col  credo  £.0230 

I Instructional  Technology  Branch 
AFHRL 

Lowry  AFB,  CO  80230 
ATTN:  Major  Brian  Waters 

1 Dr.  Alfred  R.  Frcgly 
AFOSR/M , Building  410 
Boll  in.,  AFB.  DC  20332 

1 Dr.  Sylvia  R.  Mayer  (HCITl 
HQ  Electronic  Systems  Division 
LG  Hanscom  Field 
Bedford,  HA  01730 

1 AFHRL/PED 
Stop  f63 

Lackland  AF8.  TX  78236 

1 Major  Wayne  S.  Selli.an 
Chief,  Personnel  Testing 

AFliPC/DPftYO 

Randolph  AFB.  TX  7014B 

1 Air  University  Library 
AUL/LSE  76-443 
Maxwell  AFB,  AL  36112 

Marine  Corps 

1 Director,  Office  of  Manpower 
Utilization 

IX),  Marine  Corps  (Code  MPU) 

LCD.  Building  2009 
Quantlco,  VA  22134 

1 Or.  A.  L.  Slafkosky 
Scientific  Advisor  (Code  RO-1) 
HQ,  U.S.  Marine  Corps 
Washington,  DC  20380 

Coast  Guard 


1 Or.  Harold  F.  O’Neil,  Jr. 

Advanced  Research  Projects  Agency 
Cybernetics  Technology,  Noam  (23 
1400  Ullson  Clvd. 

Arlington,  VA  22209 

1 Dr.  Robert  Young 

Advanced  Research  Project*  Agency 
ICC*)  Wilson  touievird 
Arlington,  VA  22209 


Coon.  3CP7U,  Ter  tagcin 
HiShingtcn,  DC  2C301 

12  Defense  Document* Li  on  Cenier 
Cameron  Station,  Bldg.  5 
Alexandria,  VA  22314 
Attn:  TC 

1 Military  Assistant  for  Human  Resources 
Office  of  the  Director  of  Defense 
Research  l Engineering 
Root.  3D129,  The  Pentagon 
Washington,  DC  20301' 

1 Director,  Menacerent  Information 
Sy stuns  Office 
OSD.  KARA 

Room  3D91 '/ . the  Pentagon 
Washington,  DC  203D1 


Other  Government 

1 Or.  Lorraine  0.  Eyde 
Personnel  RAO  Center 
U.S.  Civil  Service  Comission 
1900  E Street  NH 
Washington,  OC  20415 

1 Dr.  William  Gorham,  Olrector 
Personnel  RAD  Center 
U.S.  Civil  Service  Commission 
1900  E Street  HW 
Washington,  DC  20415 

1 Dr.  Vern  Urry 
Personnel  RAD  Center 
U.S.  Civil  Service  Commission 
19 CO  l Street 
Washington,  DC  20415 

1 Dr.  Andrew  R.  ttolnar 
Science  Education  Dev.  A Res. 
National  Science  Foundation 
Washington,  DC  20S50 

1 U.S.  Civil  Service  Commission 
Federal  Office  Building 
Chicago  Regional  Staff  Division 
Regional  Psychologist 
230  S.  Dearborn  Street 
Chicago,  IL  60604 
Attn:  C.  S.  Winiewicz 

1 Or.  Joseph  L.  Young,  Director 
Memory  A Cognitive  Processes 
National  Science  Foundation 
Washington,  DC  20550 

Miscellaneous 

1 Or.  John  R.  Anderson 


Now  Haven.  CT  06520 

1 Or.  Scervla  B.  Anderson 
Educational  Testing  Service 
Suite  1040 

3445  Peachtree  Road  NE 
Atlanta,  GA  30326 

I Professor  Earl  A.  Alluisl 
Cod*  287 


1 Dr.  Gerald  V.  Darrett 
University  of  Akron 
Dept,  of  Psychology 
Akron,  OH  44325 

1 Or.  Bernard  H.  Bass 
University  oi  Rochester 
Graduate  School  of  Management 
Rochester,  KY  V.627 

1 Dr.  John  Seeley  Crown 

Dolt  trranek  and  Hetman,  Inc. 

50  Moulton  Street 
Cairbridge,  I’A  021 3f! 

1 Dr.  For.ald  F.  Carver 
School  of  Education 
University  of  Missouri -Kansas  City 
5100  Rockhlll  Road 
Kansas  City.  HO  64110 

1 Century  Research  Corporation 
4113  Lee  Highway 
Arlington,  VA  22207 

1 Dr.  Kenneth  E. -Clark 
College  of  Arts  A Sciences 
University  of  Rochester 
River  Campus  Station 
Rochester,  NY  14627 

1 Dr.  Norman  Cliff 
Dept,  of  Psychology 
University  of  Southern  California 
University  Perk 
Los  Angeles,  CA  90007 

1 Dr.  Allan  H.  Collins 

Bolt  Beranek  and  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  HA  02138 

1 Dr.  John  J.  Collins 
Essex  Corporation 
6305  Caminito  Estrellado 
San  Diego,  CA  92120 

1 Dr.  Rene  V.  Bawls 
Oept.  of  Psychology 
University  of  Minnesota 
Minneapolis,  l‘N  55455 

1 Cr.  Ruth  Day 

Dept,  of  Psychology 
Yale  University 
2 Hi  11  house  Avenue 
New  Haven,  a 06520 


1 Cr.  Marvin  0.  Dunnette 
Dept,  of  Psychology 
Univeriity  of  Minnesota 
Minneapolis,  MN  55455 

1 ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Dethesde,  MO  20014 

1 Major  I.  N.  Evonic 

Canadian  Forces  Personnel 
Applied  Research  Unit 
1107  Avenue  Road 
Toronto,  Ontario,  CANADA 


Pep*,  of  Psychology 

Old  Dominion  University 
Norfolk,  VA  235C3 


1 Hr.  Joseph  J.  Cowan,  Chief 
Psychological  Research  Branch  (6-P-1/62] 
U.S.  Coast  Guard  Headquarters 
Washington,  OC  20S90 


! 


1 Dr.  Richard  L.  Ferguson 

The  finer  lean  College  Testing  Program 

P.  0.  Pox  168 

Iowa  City,  IA  52240 

1 Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  HD  20850 

1 Dr.  Edwin  A.  Fleishman 

Advanced  Research  Resources  Organization 
8555  Sixteenth  Street 
Silver  Spring,  HO  20910 

1 Dr.  John  R.  Treiierlksen 
Dolt  Beranck  & Keenan,  Inc. 

50  Houlton  Street 
Cambridge,  I1A  02138 

1 Dr.  Robert  Glaser,  Co-Director 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1 Dr.  H.  D.  Havron 

Hunan  Sciences  Research,  Inc. 

7710  Cld  Spring  House  Road 
Uest  Crte  Industrial  Park 
McLean,  VA  22101 

1 Or.  Duncan  Hansen 
School  of  Education 
Memphis  State  University 
Memphis,  TN  38118 

1 Human  Resources  Research  Organization 
400  Plaza  Bldg. 

Pace  Blvd.  at  Fairfield  Drive 
Pensacola,  FL  32505 

1 Huml’.RO/Westem  Division 
27857  Berwick  Drive 
Camel.  CA  93921 
Attn:  Library 

1 HunRRO/ Columbus  Office 

Suite  23,  2601  Cross  Country  Drive 
Columbus,  GA  31906 

1 Dr.  Lawrence  8.  Johnson 

Lawrence  Johnson  I Associates,  Inc. 

Suite  S02 

2001  S Street  NU 

Washington,  DC  20009 

1 Or.  Roger  A.  Kaufman 
203  Dodd  Hall 
Florid?  State  University 
Tall  alias  ses,  FL  323C6 

I Dr.  Steven  N.  Keete 
Dept,  of  Psychology 
University  of  Oregon 
Eugene,  OR  97403 

1 Or.  Devld  Klahr 
Oept.  of  Psychology 
Carnegie -Me lion  University 
Pittsburgh,  PA  15213 

1 Dr.  Alma  E.  Lantz 
University  of  Denver 
Denver  Research  Institute 
Industrial  Economics  Division 
Denver,  CO  80210 

1 Mr.  U.  E.  Lassiter 
Oata  Solutions  Corp. 

Suite  211,  6849  Old  Dominion  Drive 
McLean,  VA  22101 


1 Dr.  Frederick  M.  Lord  . 1 

Educational  Testing  Service 
Princeton,  NJ  08540 

1 Mr.  Brian  McNally 

Educational  lestlnq  Service  ■ 

Princeton,  NJ  08540 

1 Dr.  Ernest  J.  McCormick 

Department  of  Psychological  Sciences 
Purdue  University 

Lafayette,  IN  47907  1 

1 Dr.  Robert  R.  Hackle 

Human  Factors  Research,  Inc. 

6780  Corton  Drive 

Santa  Barbara  Research  Park 

Goleta,  CA  93017  \ 

1 Dr.  William  C.  Hann 

University  of  So.  California 
Information  Sciences  Institute 
4676  Admiralty  Way  1 

Marina  Del  Roy,  CA  90291 

1 Mr.  Fell  o:id  ."arks 

304  Granye  Bldg. 

Pennsylvania  State  University  1 

University  Park,  PA  16802 

1 Dr.  Leo  Munday 
Houghton  Mifflin  Co. 

P.  0.  Box  1970  , 

I ewe  City,  IA  52240  1 

1 Richard  T.  Mowday 

College  of  Business  Administration 
University  of  Nebraska,  Lincoln  . 

Lincoln,  NE  68588  ' 

1 Dr.  Donald  A.  Norman 
Dept,  of  Psychology  1-009 
University  of  California,  San  Diego 
La  Jolla,  CA  92093  1 

1 Mr.  Luigi  Petrol lo 
2431  N.  Edgewood  Street 
Arlington.  VA  22207 

1 Dr.  Lyman  H.  Porter,  Dean 

Graduate  School  of  Administration 
University  of  California 
Irvine.  CA  92717 

1 Dr.  Diane  M.  Ramsey-Klee 
R-K  Research  A System  Design 
3947  Rldgemont  Drive 
Malibu.  CA  98265 

1 R.DIr.  11.  Rauch 
P II  4 

Bundesmlnlsterlum  dor  Vorteldlgung 
Postfach  HI 
53  Bonn  1 , GERMANY 

1 Dr.  Joseph  W.  Rlgney 

University  of  So.  California 
Behavioral  Technology  Laboratories 
3717  South  Grand 
Los  Angeles,  CA  90007 

1 Dr.  Andrew  N.  Rose 

American  Institutes  for  Research 
105S  Thornes  Jefferson  St.  NW 
Washington,  DC  20007 

1 Dr.  Leonard  L.  Rosenbaum,  Chairman 
Dept,  of  Psychology 
Montgomery  College 
Rockville.  MO  20350 


Dr.  Benjamin  Schneider 
Dept,  of  Psychology 
University  of  Maryland 
College  Park.  MO  20742 

Dr.  Mark  bckiH 

Uni  varsity  of  HUaourl-Colwbla 

Dept,  of  Muc.  Psychology 

12  >111  hall 

Coliabla,  MO  65201 

Dr.  Robert  J.  Seidel 
Instructional  Technology  Group, 
HumT.R0 

300  N.  Washington  St. 
Alexandria,  VA  22314 

Dr.  Richard  Snow 
Stanford  University 
School  of  Education 
Stanford , CA  94305 

Mr.  Dennis  J.  Sulllvtr. 
c/o  Canyon  Research  Group,  Inc. 
32107  Llndcro  Car, yon  Road 
Westlake  Village,  CA  9136b 

Or.  Keith  Wescourt 
Dept,  of  Psychology 
Stanford  University 
Stanford,  CA  94305 

Dr.  Anita  West 
Denver  Research  Institute 
University  of  Denver 
Oenver,  CO  80201 

Dr.  Earl  Hunt 
Dept,  of  Psychology 
University  of  Washington 
Seattle,  WA  92105 

Or.  Thomas  G.  Stlcht 
Assoc.  Director,  Basic  Skills 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington.  OC  20208 


1 


